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EXECUTIVE  SUMMARY 

EVALUATION  OF  THE  EFFECTIVENESS  OF 
VOCATIONAL  SYSTEMS  IN  ACHIEVING  THEIR  PURPOSES 

The  State  Council  on  Vocational  Education  conducts  biennial 
evaluations  of  vocational  education  systems  supported  with  federal  funds. 
These  systems  include  vocational  schools,  comprehensive  high  schools, 
community  colleges,  and  community-based  organizations.  The  1987  evaluation, 
performed  for  the  State  Council  by  Abt  Associates  Inc.,  addresses  three 
issues : 

•  What  are  the  stated  goals  of  vocational  education  in 
Massachusetts? 

•  How  do  state  policymakers  perceive  vocational 
education  and  what  are  their  goals  for  vocational 
education? 

•  In  the  view  of  local  vocational  education  providers, 
which  goals  are  important  and  how  well  are  those 
goals  implemented? 

The  first  and  second  questions  consider  expectations  for  and  about  vocational 
education  in  Massachusetts  as  well  as  the  commonality  of  those  expectations, 
while  the  third  explores  the  extent  to  which  vocational  education  is  deemed  to 
be  meeting  its  stated  objectives. 

A  three-part  study  was  designed  to  answer  these  questions.  The 
first  study  task  was  a  review  of  goals  presented  in  a  variety  of  documents 
including  the  Carl  D.  Perkins  Vocational  Education  Act,  Massachusetts 
Chapter  74,  the  State  Plan  for  Vocational  Education  in  Massachusetts,  the 
Massachusetts  Training  and  Employment  Plan  for  the  Job  Training  Partnership 
Act  (JTPA),  the  State  Board  of  Education's  Comprehensive  Policy  on 
Occupational  Education,  the  Board's  Policy  Statement  on  Public  School  Youth, 
Education  and  Employment,  and  the  Joint  Policy  on  Occupational  Education 
prepared  by  the  State  Board  of  Education  and  Board  of  Regents. 

Interviews  with  state  policymakers  comprised  the  second  part  of  the 
study.  Thirteen  state  policymakers  who  influence  policy  and/or  funding  for 
vocational  education  were  identified  by  the  State  Vocational  Education 
Council.  The  list  of  interviewees  was  not  all  inclusive  but  representative  of 
individuals  who  could  play  central  roles.   They  were  interviewed  to  obtain 


their  views  of  important  vocational  education  goals  and  their  recommendations 
for  change  to  the  existing  vocational  education  system. 

The  third  study  task  was  a  telephone  survey  of  local  providers  of 
vocational  education  and  professionals  in  related  fields.  Interviews  were 
conducted  with  approximately  fifty  respondents  in  ten  Massachusetts 
communities.  The  survey  asked  respondents  to  rank  the  importance  of  goals 
obtained  from  various  documents  and  policymaker  interviews  and  to  indicate  how 
well  the  Massachusetts  vocational  education  system  is  meeting  these  goals. 
They  were  also  asked  to  rank  the  importance  of  a  series  of  system 
improvements. 

To  summarize,  it  was  necessary  to  identify  and  list  the  goals  of  the 
Massachusetts  vocational  education  system  in  order  to  evaluate  the  system  in 
terms  of  its  relative  purposes.  The  views  of  policymakers  were  regarded  as 
important  because  they  might  put  forth  additional  goals.  Also,  policymaker 
perceptions  of  goal  importance  and  achievement  were  reviewed  to  understand  and 
explicate  the  extent  of  agreement  between  state  policymakers  and  local 
providers.  The  goals  statements  in  the  local  provider  questionnaire  were 
based  on  information  obtained  during  the  document  review  as  well  as  on 
information  drawn  from  state  policymaker  interviews. 

It  is  important  to  point  out  an  unusual  feature  of  the  evaluation 
described  here.  This  study  looks  at  perceptions  of  system-wide  goal 
importance  and  attainment  rather  than  effectiveness  of  individual  programs. 
It  is  concerned  with  subjective  assessments.  It  does  not  focus  on  so-called 
objective  measures  of  program  achievement  such  as  student  test  scores  or 
employer  satisfaction. 

Selected  major  findings  of  these  inquiries  are  summarized  below. 

Review  of  Written  Goals.  The  consideration  of  state  and  federal  laws  and 
regulations  and  state  plans  and  policy  statements  produced  the  following 
results: 

•  A  multiplicity  of  goals  for  vocational  education.  The 
goals,  listed  in  these  documents,  can  be  categorized 
into  five  topical  areas:  student  outcomes,  equal  access 
provisions,  economic  development,  system  improvement, 
and  system  coordination. 


•  A  Lack  of  established  priorities  for  the  multiple  goals 
set  forth  by  the  various  laws,  plans,  and  policy 
statements . 

•  Only  one  comprehensive  statement  of  goals — the  State 
Plan  for  Vocational  Education  in  Massachusetts.  The 
plan  is  produced  for  federal  compliance  purposes  and 
generally  includes  the  goals  of  other  state  documents. 

•  Chapter  74  regulations  briefly  state  student  outcome 
goals  but  are  mainly  concerned  with  program  approval 
criteria.  Chapter  74  focuses  primarily  on  the  estab- 
lishment, organization,  and  governance  of  vocational 
schools  and  programs.  It  also  deals  substantially  with 
the  certification  of  personnel. 

Interviews  with  State  Policymakers.   Information  obtained  includes: 

•  Substantial  agreement  that,  at  the  secondary  school 
level,  vocational  education  should  concentrate  on 
providing  a  comprehensive  education,  buttressed  with 
career  exploration  and  learning  in  work  settings.  All 
policymakers  saw  specific  skills  training  as  the  most 
important  goal  of  postsecondary  vocational  education, 
while  skills  training  at  the  secondary  level  was  given  a 
mixed  response. 

•  Noticeable  differences  in  the  emphasis  on  and  proposed 
methods  for  increasing  participation  in  the  vocational 
education  system  by  unserved  or  underserved  groups. 
Half  of  the  policymakers  strongly  criticized  vocational 
education  for  serving  too  narrow  a  group  and  for  not 
taking  more  affirmative  steps,  while  the  second  half 
were  less  concerned  with  this  issue  or  questioned  the 
efficiency  or  consequences  of  targeting. 

•  Clear  agreement  that  vocational  education's  primary  role 
in  economic  development  is  adult  retraining  and  skills 
upgrading. 

•  A  lack  of  consensus  on  needed  vocational  education 
system  improvement. 

•  Strong  emphasis  on  the  lack  of  and  resulting  need  for 
state  and  local  system  coordination,  particularly 
between  vocational  education  and  employment  and  training 
programs. 


Local-Level  Telephone  Survey.   Responses  to  the  telephone  survey  include: 

•  Remarkable  agreement  on  the  importance  of  goals  for  both 
secondary  and  postsecondary  education.  In  fact,  four  of 
the  five  goals  ranked  as  most  important  for  secondary 


vocational  education  were  aLso  identified  as  four  of  the 
five  most  important  goals  for  postsecondary  education. 
These  goals  are:  enhancing  students'  career 
decisionmaking;  ensuring  access  to  vocational  education 
for  unserved  and  underserved  groups;  providing  hands-on 
learning  for  students;  and  teaching  students  specific 
occupational  skills. 

•  Virtually  complete  consensus  among  respondents  that,  for 
both  secondary  and  postsecondary  vocational  education, 
the  Massachusetts  vocational  system  is  not  achieving 
most  of  its  stated  goals.  An  exception  to  this  finding 
is  that  between  60  and  70  percent  of  the  local 
respondents  felt  that  the  Massachusetts  secondary-level 
vocational  education  system  achieves  two  goals: 
teaching  students  specific  occupational  skills  and 
providing  "hands-on"  learning  for  students. 

•  Agreement  between  local  respondents  and  state 
policymakers  that  secondary  vocational  education  should 
provide  students  with  a  comprehensive  education 
(ensuring  basic  skills). 

•  Local  respondents  were  much  more  likely  than  state 
policymakers  to  see  specific  skills  training  as  an 
important  goal  for  secondary  vocational  education. 

•  Agreement  among  local  respondents  that  postsecondary 
vocational  education  should  provide  students  with 
specific  occupational  skills  and  create  a  skilled 
workforce . 

•  Clear  consensus,  unlike  the  disparate  views  expressed  by 
state  policymakers,  on  the  importance  of  ensuring  access 
to  the  vocational  education  system  to  a  variety  of 
special  populations. 

•  A  clustering  of  needed  improvements  for  vocational 
education  in  Masscahusetts.  Between  85  and  90  percent 
of  local  respondents  identified  the  following  system 
improvements:  coordination  between  vocational  education 
and  employment  and  training  programs;  stronger  local 
linkages  with  other  educational  programs;  increased 
funding;  higher  quality  of  professional  staff 
development;  and  development  of  opportunities  for 
students  to  have  work-related  experience. 

•  Agreement  between  state  policymakers  and  local  providers 
that  system  coordination  is  an  important  goal  of  and 
necessary  improvement  for  the  Massachusetts  vocational 
education  system.  Local  providers  felt  that  state-level 
coordination  was  poorly  achieved  or  not  achieved  at  all. 


